<HTML>
<HEAD>
<!-- This HTML file has been created by texi2html 1.29
     from lex.tnf on 19 December 2010 -->

<TITLE>Lexical Analysis - Literal Symbols</TITLE>
</HEAD>
<BODY TEXT="#000000" BGCOLOR="#FFFFFF" LINK="#0000EE" VLINK="#551A8B" ALINK="#FF0000" BACKGROUND="gifs/bg.gif">
<TABLE BORDER=0 CELLSPACING=0 CELLPADDING=0" VALIGN=BOTTOM>
<TR VALIGN=BOTTOM>
<TD WIDTH="160" VALIGN=BOTTOM>
<A HREF="http://eli-project.sourceforge.net/">
<IMG SRC="gifs/elilogo.gif" BORDER=0>
</A>&nbsp;
</TD>
<TD WIDTH="25" VALIGN=BOTTOM>
<img src="gifs/empty.gif" WIDTH=25 HEIGHT=25>
</TD>
<TD ALIGN=LEFT WIDTH="475" VALIGN=BOTTOM>
<A HREF="index.html"><IMG SRC="gifs/title.png" BORDER=0></A>
</TD>
<!-- |DELETE FOR SOURCEFORGE LOGO|
<TD>
<a href="http://sourceforge.net/projects/eli-project">
<img
  src="http://sflogo.sourceforge.net/sflogo.php?group_id=70447&amp;type=13"
  width="120" height="30"
  alt="Get Eli: Translator Construction Made Easy at SourceForge.net.
    Fast, secure and Free Open Source software downloads"/>
</a>
</TD>
|DELETE FOR SOURCEFORGE LOGO| -->
</TR>
</TABLE>

<HR size=1 noshade width=785 align=left>
<TABLE BORDER=0 CELLSPACING=2 CELLPADDING=0>
<TR>
<TD VALIGN=TOP WIDTH="160">
<h4>General Information</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="index.html">Eli: Translator Construction Made Easy</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="gindex_1.html#SEC1">Global Index</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="faq_toc.html" >Frequently Asked Questions</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ee.html" >Typical Eli Usage Errors</a> </td></tr>
</table>

<h4>Tutorials</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="EliRefCard_toc.html">Quick Reference Card</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="novice_toc.html">Guide For new Eli Users</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="news_toc.html">Release Notes of Eli</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="nametutorial_toc.html">Tutorial on Name Analysis</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="typetutorial_toc.html">Tutorial on Type Analysis</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ee.html" >Typical Eli Usage Errors</a> </td></tr>
</table>

<h4>Reference Manuals</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ui_toc.html">User Interface</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="pp_toc.html">Eli products and parameters</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="lidoref_toc.html">LIDO Reference Manual</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ee.html" >Typical Eli Usage Errors</a> </td></tr>
</table>

<h4>Libraries</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="lib_toc.html">Eli library routines</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="modlib_toc.html">Specification Module Library</a></td></tr>
</table>

<h4>Translation Tasks</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="lex_toc.html">Lexical analysis specification</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="syntax_toc.html">Syntactic Analysis Manual</a></td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="comptrees_toc.html">Computation in Trees</a></td></tr>
</table>

<h4>Tools</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="lcl_toc.html">LIGA Control Language</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="show_toc.html">Debugging Information for LIDO</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="gorto_toc.html">Graphical ORder TOol</a> </td></tr>
</table>
<p>
<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="fw_toc.html">FunnelWeb User's Manual</a> </td></tr>
</table>
<p>
<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="ptg_toc.html">Pattern-based Text Generator</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="deftbl_toc.html">Property Definition Language</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="oil_toc.html">Operator Identification Language</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="tp_toc.html">Tree Grammar Specification Language</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="clp_toc.html">Command Line Processing</a> </td></tr>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="cola_toc.html">COLA Options Reference Manual</a> </td></tr>
</table>
<p>
<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="idem_toc.html">Generating Unparsing Code</a> </td></tr>
</table>
<p>
<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="mon_toc.html">Monitoring a Processor's Execution</a> </td></tr>
</table>

<h4>Administration</h4>

<table BORDER=0 CELLSPACING=0 CELLPADDING=0>
<tr valign=top><td><img src="gifs/gelbekugel.gif" WIDTH=7 HEIGHT=7 ALT=" o"> </td><td><a href="sysadmin_toc.html">System Administration Guide</a> </td></tr>
</table>

<HR WIDTH="100%">
<A HREF="mailto:eli-project-users@lists.sourceforge.net">
<IMG SRC="gifs/button_mail.gif" BORDER=0 ALIGN="left"></A>
<A HREF="index.html"><IMG SRC="gifs/home.gif" BORDER=0 ALIGN="right"></A>

</TD>
<TD VALIGN=TOP WIDTH="25"><img src="gifs/empty.gif" WIDTH=25 HEIGHT=25></TD>

<TD VALIGN=TOP WIDTH="600">
<H1>Lexical Analysis</H1>
<P>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_3.html"><IMG SRC="gifs/prev.gif" ALT="Previous Chapter" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_5.html"><IMG SRC="gifs/next.gif" ALT="Next Chapter" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_toc.html"><IMG SRC="gifs/up.gif" ALT="Table of Contents" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT="">
<HR size=1 noshade width=600 align=left>
<H1><A NAME="SEC20" HREF="lex_toc.html#SEC20">Literal Symbols</A></H1>
<P>
If the generated processor includes a parser
(see  <A HREF="syntax_toc.html">Top of Syntactic Analysis</A>),
then Eli will extract the descriptions of any literal terminal symbols from
the context-free grammar defining that parser and add them to the
specifications provided by type-<TT>`gla'</TT> files.
For example, consider the following context-free grammar:
<P>
<PRE>
Program: Expression .
Expression: Evaluation / Binding .
Evaluation:
  Constant / BoundVariable /
  '(' Expression '+' Expression ')' /
  '(' Expression '*' Expression ')' .
Binding: 'let' BoundVariable '=' Evaluation 'in' Expression .
</PRE>
<P>
This grammar has nine terminal symbols.
Two (<CODE>Constant</CODE> and <CODE>BoundVariable</CODE>) are given by identifiers,
and the other seven (<CODE>(</CODE>, <CODE>)</CODE>, <CODE>+</CODE>, <CODE>*</CODE>, <CODE>let</CODE>,
<CODE>=</CODE> and <CODE>in</CODE>) are given by literals.
<P>
Only the character sequences to be classified as <CODE>Constant</CODE> or
<CODE>BoundVariable</CODE>, and those to be classified as comments, need be
defined by type-<TT>`gla'</TT> files.
Descriptions of the symbols given as literals will be automatically
extracted from the grammar by Eli.
Thus the lexical analyzer for this language might be described by a single
type-<TT>`gla'</TT> file containing the following:
<P>
<PRE>
Constant:      PASCAL_INTEGER
BoundVariable: PASCAL_IDENTIFIER
               PASCAL_COMMENT
</PRE>
<P>
<H2><A NAME="SEC21" HREF="lex_toc.html#SEC21">Overriding the Default Treatment of Literal Symbols</A></H2>
<P>
By default, a literal terminal symbol specified in a context-free grammar
supplied to Eli will be recognized as though it had been specified by the
appropriate regular expression.
Thus the literal symbols <CODE>'+'</CODE> and <CODE>'let'</CODE> will be recognized as
though the following specifications had been given by the user:
<P>
<PRE>
Plus:  $\+
Let:   $let
</PRE>
<P>
(Here <CODE>Plus</CODE> and <CODE>Let</CODE> are arbitrary identifiers describing the
initial classifications of the literal symbols.
No such identifiers are actually supplied by Eli, but the literal symbols
are <EM>not</EM> initially classified as comments.)
<P>
In some situations it is useful to carry out more complex operations at the
time the literal symbol is recognized.
In this case, the user must do two things:
<P>
<OL>
<LI>
Mark the literal symbol as being a special case.
<P>
<LI>
Provide a specification for the literal symbol.
</OL>
<P>
As a concrete example, suppose that <CODE>%%</CODE> were used as a major separator
in the input text and could appear either once or twice.
Assume that the first occurrence is required, and the second is optional.
All text following the second occurrence is to be ignored.
<P>
One approach to this problem would be to count the number of occurrences of
the literal symbol <CODE>%%</CODE>, advancing to the end of the input text after
the second.
This could be done by an auxiliary scanner (see  <A HREF="lex_1.html#SEC7">Auxiliary Scanners</A>)
that either returns a pointer to the character following the <CODE>%%</CODE> or
a pointer to the ASCII NUL terminating the input text,
and a token processor (see  <A HREF="lex_1.html#SEC10">Token Processors</A>)
that reclassifies the second occurrence of <CODE>%%</CODE> as a comment.
The grammar would specify only the required first occurrence of <CODE>%%</CODE>.
<P>
In order to mark the literal symbol <CODE>%%</CODE> as a special case that
should not receive the default treatment, the user must supply a
type-<TT>`delit'</TT> file specifying that symbol as a regular expression.
The entry in the type-<TT>`delit'</TT> file also needs to define an identifier
to represent the classification:
<P>
<PRE>
$%%  PercentPercent
</PRE>
<P>
Each line of a type-<TT>`delit'</TT> file consists of a regular expression and
an identifier, separated by white space.
The regular expression must describe a literal symbol appearing in a
context-free grammar supplied to Eli.
That literal symbol will not be incorporated automatically into the
generated lexical analyzer; it must be specified explicitly by the user.
The identifier will be given the appropriate value by an Eli-generated
<CODE>#define</CODE> directive in file <TT>`litcode.h'</TT>.
<P>
In our example, <CODE>%%</CODE> could be specified by the following line of a
type-<TT>`gla'</TT> file:
<P>
<PRE>
  $%%  (SkipOrNot) [CommentOrNot]
</PRE>
<P>
Initially, the separator will be classed as a comment because there is no
identifier preceding the regular expression.
<CODE>SkipOrNot</CODE> will use a state variable to decide whether or not to skip
text (see  <A HREF="lex_1.html#SEC9">Building scanners</A>),
while <CODE>CommentOrNot</CODE> will use the same state variable to decide
whether or not to change the classification to <CODE>PercentPercent</CODE>
(see  <A HREF="lex_1.html#SEC12">Building processors</A>):
<P>
<PRE>
#include &#60;fcntl.h&#62;
#include "source.h"
#include "litcode.h"

static int Second = 0;

char *
SkipOrNot(char *start, int length)
{ if (!Second) return start + length;
  (void)close(finlBuf());
  initBuf("/dev/null", open("/dev/null", O_RDONLY));
  return TEXTSTART;
}

void
CommentOrNot(char *start, int length, int *syncode, int *intrinsic)
{ if (!Second) { Second++; *syncode = PercentPercent; }
}
</PRE>
<P>
The remainder of the text is skipped by closing the current input file and
opening an empty file to read
(see  <A HREF="lib_1.html#SEC3">Text Input of Library Reference Manual</A>).
Since <CODE>%%</CODE> is initially classified as a comment, its classification
must be changed only on the first occurrence.
<P>
File <TT>`fcntl.h'</TT> defines <CODE>open</CODE> and <CODE>O_RDONLY</CODE>,
<TT>`source.h'</TT> defines <CODE>initBuf</CODE>, <CODE>finlBuf</CODE> and
<CODE>TEXTSTART</CODE>,
and <TT>`litcode.h'</TT> defines <CODE>PercentPercent</CODE>.
<P>
<H2><A NAME="SEC22" HREF="lex_toc.html#SEC22">Using Literal Symbols to Represent Other Things</A></H2>
<P>
In some cases the phrase structure of a language depends upon
layout cues rather than visible character sequences.
For example, indentation is used in Occam2 to indicate block structure:
If the first non-blank character of a line is indented further than the
first non-blank character of the line preceding it,
then the new line begins a new block.
If the first non-blank character of a line is not indented as far as the
first non-blank character of the line preceding it, then the old line ends
one or more blocks depending on the difference in indentation.
If the first non-blank characters of two successive lines are indented by
the same amount, then the lines simply contain adjacent statements of the
same block.
<P>
Layout cues can be represented by literal symbols in the context-free grammar
that describes the phrase structure.
The processing needed to recognize the layout cues can then be described in
any convenient manner, and the sequence of white space characters implementing 
those cues can be classified as the appropriate literal symbol.
<P>
Suppose that the beginning of a block is represented in the Occam2 grammar
by the literal symbol <CODE>'{'</CODE>, the statement separator by <CODE>';'</CODE>,
and the end of a block by <CODE>'}'</CODE>.
In the input text, blocks and statement separators are defined by layout
cues as described above.
A type-<TT>`delit'</TT> file marks the literal symbols as requiring special
recognition and associates an identifier with each:
<P>
<PRE>
$\{  Initiate
$;  Separate
$\}  Terminate
</PRE>
<P>
Indentation can be specified as white space following a new line:
<P>
<PRE>
  $\n[\t\040]*  [OccamIndent]
</PRE>
<P>
The token processor <CODE>OccamIndent</CODE> would carry out all of the processing
necessary to determine the meaning of the indentation.
This processing is complex, involving interactions with several other
components of the generated lexical analyzer
(see  <A HREF="lex_6.html#SEC33">An Example of Interface Usage</A>).
It constitutes an operational definition of the meaning of indentation in
Occam2.
<P>
<HR size=1 noshade width=600 align=left>
<P>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_3.html"><IMG SRC="gifs/prev.gif" ALT="Previous Chapter" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_5.html"><IMG SRC="gifs/next.gif" ALT="Next Chapter" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT=""><A HREF="lex_toc.html"><IMG SRC="gifs/up.gif" ALT="Table of Contents" BORDER="0"></A>
<IMG SRC="gifs/empty.gif" WIDTH=25 HEIGHT=25 ALT="">
<HR size=1 noshade width=600 align=left>
</TD>
</TR>
</TABLE>

</BODY></HTML>
